I tested this in a VM and it works. However, one thing I noticed that's lost is the repo pruning. The deployment is now written out in the shutdown path and ostree makes the decision not to trigger pruning there. I think we want to add a call to ostree_sysroot_cleanup_prune_repo somewhere, but I don't know where yet. I'm going to see where they do this in rpm-ostree.

Hmm, CI hit a test timeout, which I'd also seen in OBS. There's currently a 360 second timeout for the slower tests and I routinely see the flatpak install test take nearly 300 seconds on my fast laptop. I think I'll add a commit to bump that to 600 seconds.

More detail in phabricator, but for pruning it appears that rpm-ostree just does the normal cleanup once the deployment is staged. Since the rollback deployment hasn't been deleted at that point (it's deleted when finalizing), you don't get the disk space back immediately. That would happen the next time you apply an update (with the current code).

If we want to be more aggressive about reclaiming the disk space, we'd have to add another ostree_sysroot_cleanup in an earlier state. One thought I had was to do it when polling indicates no updates. But that could thrash the disk unnecessarily often.

An idea I haven't thought through:

• eos-updater-cleanup.service, which (calls a D-Bus method on eos-updater which) runs ostree_sysroot_cleanup, and is disabled by default
• eos-updater-trigger-cleanup.service, enabled with ConditionNeedsUpdate=/etc, which just runs systemctl enable --runtime eos-updater-cleanup.service

I'm sure the answer is "not easily" but can the Flatpak install test be made to take less than 300 seconds, perhaps with a crude implement such as eatmydata or running entirely on tmpfs?

I'm sure the answer is "not easily" but can the Flatpak install test be made to take less than 300 seconds, perhaps with a crude implement such as eatmydata or running entirely on tmpfs.

I would like that since the test suite turnaround is slow for development. I need to figure out where it's taking so much time. I added the commit here which we can banter about. I also punted on the more aggressive pruning, which we can also discuss.

An idea I haven't thought through:

• eos-updater-cleanup.service, which (calls a D-Bus method on eos-updater which) runs ostree_sysroot_cleanup, and is disabled by default
• eos-updater-trigger-cleanup.service, enabled with ConditionNeedsUpdate=/etc, which just runs systemctl enable --runtime eos-updater-cleanup.service

I like this. It could be even simpler since ostree admin cleanup is just a wrapper around ostree_sysroot_cleanup. Any reason not to eos-updater-cleanup.service have the ConditionNeedsUpdate=/etc directly?

My thinking was that don't want it to block the boot process, c.f. eos-split-flatpak-repos.service, so it wouldn't be triggered early enough. But perhaps it could be Type=simple so systemd doesn't wait for it to finish. Add a sprinkling of IOSchedulingClass=idle, ...

My thinking was that don't want it to block the boot process, c.f. eos-split-flatpak-repos.service, so it wouldn't be triggered early enough. But perhaps it could be Type=simple so systemd doesn't wait for it to finish. Add a sprinkling of IOSchedulingClass=idle, ...

Ah, right. Especially since this has to run Before=systemd-update-done.service, so it definitely would block early boot.

Another idea that's a little simpler to avoid ConditionNeedsUpdate is to use a separate stamp file. After all, eos-updater knows exactly when the system is updated. For instance, it could create /sysroot/.cleanup and then we would have a systemd unit like so:

[Unit]
ConditionPathExists=/sysroot/.cleanup

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/ostree admin cleanup
ExecStartPost=/bin/rm -f /sysroot/.cleanup

In order to not get it to block multi-user.target, you could use path unit. Or we could just set DefaultDependencies=no so that multi-user.target doesn't automatically add After to it.

I actually think the above would be useful upstream as a complement to the existing ostree-finalize-staged.service.

Perhaps it should put the stamp file into /etc in the new deployment? Otherwise, consider these steps:

1. Boot into deployment A
2. Stage deployment B, write stamp file
3. Unclean shutdown
4. Boot into deployment A
5. ostree admin cleanup runs, stamp file removed
6. Shut down cleanly. This causes B to be finalized
7. Boot into B. No cleanup

Good point about /etc and an unclean shutdown. I opened ostreedev/ostree#2510 upstream. If the file was written out by the ostree finalization routine, then it could be in /sysroot.

It looks like ostreedev/ostree#2510 is probably not imminent, so I'm going to do something similar downstream. I think I'm going to try to address it here, but if I get stuck I think we can merge this and fix the pruning in a follow up.

I added a commit to handle cleanup with a drop-in for ostree-finalize-staged.service + a new eos-updater-autocleanup.service unit. For the latter it's just running ostree admin cleanup and rm -f /sysroot/.cleanup, but I could also write a small utility in eos-updater that handles /sysroot/.cleanup itself.

I tested this in a VM and it seems to DTRT. On my SSD system it took 5 seconds and didn't block gdm:

Jan 11 11:22:14 endless systemd[1]: Starting Automatically cleanup after staged Endless OS Updater deployment...
...
Jan 11 11:22:18 endless systemd[1]: Reached target Multi-User System.
Jan 11 11:22:18 endless systemd[1]: Reached target Graphical Interface.
...
Jan 11 11:22:19 endless systemd[1]: Finished Automatically cleanup after staged Endless OS Updater deployment.

Something I thought of later is that the ostree_sysroot_cleanup run after the deployment is initially staged has basically no effect with the addition of the autocleanup handling. In particular, it will run a repo prune that is unlikely to prune anything since the refs for the old rollback deployment haven't been deleted yet. I added a comment that it could be skipped in that case. There's no harm to keeping it except that there's a process hammering your disk for likely no gain.